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ABSTRACT 



An image processing technique in which illumination 
effects are reduced by subtraction of spatially weighted 
masks which are formed substantially simultaneously on 
an analog smoothing grid. Haloing are reduced by using 
a mask whose spatial weighting is non-linear at the 
boundary of image segments whose intensities differ by 
more than would be predicted by the Rule of Twenty 
which predicts that the ratio of naturally found reflec- 
tances seldom exceeds a factor of twenty. A color 
image is processed by forming differently scaled masks, 
each of which is related to a spectral component of the 
original image. The masks are subtracted from the 
image to form a composite image. The saturation level 
in each smoothing grid is selected to minimize adverse 
effects of illumination in that spectral component by 
causing segmentation of image areas whose intensity 
differences exceed natural reflectance ratios. The trans- 
fer functions of each grid may also separately adjusted 
to enhance the composite image. 

14 Claims, 12 Drawing Sheets 
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called color or brightness constancy, range-matching or 
SATURABLE SMOOTHING GRID FOR IMAGE enhancement. 

PROCESSING Many algorithms for determining color constancy 

corrections have been developed which attempt to pro- 
BACKGROUND OF THE INVENTION 5 vide information for correcting the colors at each point 

1. Field of the Invention m a 00101 """S 6 m ^vousc to color average informa- 
This invention relates to image processing and in tion from surrounding points. In general, such algo- 

particular to techniques for processing images captured rithms dictate the reduction of a particular color at a 
under varying colors, and/or levels of illumination, specific point if surrounding points indicate an over- 
such as video imaging systems. 10 abundance of that color. That is, a spatially weighted 

2. Description of the Prior Art average of color is removed from each specific point. 

Conventional image processing systems capture and For example, in a color image photographed at sun- 
reproduce images as illuminated by ambient light. That set, an overabundance of red would be detected and the 
is, an image will be reproduced differently if captured ■ algorithm used would indicate that the red color at each 
under different ambient illumination conditions. Such 15 point be reduced by the addition of a determinable 
conventional systems may have difficulty reproducing intensity of red*s complementary color, green, at that 
the colors, and/or levels of intensity and contrast, of an point The resultant corrected image would appear 
image as seen by a human observer as a result of the more like an image photographed in midday sunlight, 
color of the illumination and/or varying levels of illumi- Conventional techniques for implementing such 
nation in the image. 20 image color constancy correction algorithms require 

For example, the skin tones of a color image of a substantial time for the necessary mathematical calcula- 

person photographed in midday sunlight will differ tions and therefore must be performed during post pro* 

from the skin tones of a photograph of the same person cessing if at all. Although post processing may be possi- 

taken at sunset. The ambient light at sunset includes y e f or images, conventional techniques are very 
colors which are reflected by the skin and reproduced 25 tcdious ^ timc consuming and therefore little used, 

in the skin tones by the imaging process. To a human when used, such conventional color constancy correc- 

viewer, however, the skin tones appear about the same, tion systcms typically require point by point color cor- 

whether viewed at sunset or in midday sunlight as a rections . video imaging systems require almost con- 
result of automatic corrections made by the human ^ stant corrcctions ^ such conventional algorithm im- 

ncrvous system. plementation techniques are not convenient for com- 

The eye and brain, however, are thought to be able to mQn even durm ^ processing< 
perceive the approximate reflectance of objects under a corTCCtion techniques also 

variety of uluminants, colored and achromatic. These noticcabie ^ of 

abiliUes » <^ M ~ 35 incorrect coloration* at color transition edges. That is, 

constancy. These constancies result from range com- 35 techniques rely on spatial color 

pression, with the range maximum set by the bnghtest ; * " ^lIT^iZ 'IcLJL. tL ; 

It most colorful objecTin a portion of the scene. As the ™m 

illumination varies, the light reaching the eye from the erly corrected in the vicuuty of the. *&<* *ot e^ple 

most reflective object varies, and the range is adjusted * WB on a gray back ^<™^ 

accordingly 40 noticeable green halo near the edge of the ball because 

It is thought that while the sensors of the eye can the average color in the vicinity of the color transition 

faithfully captureimageintensitiesoverarangeofmore ^ not accurately reflect, on average, the overabun- 

than three orders of magnitude, the range of the optic dancc of illumination coloration, 

nerve, which carries the image from the retina in the ™ e human central nervous system, on the other 

eye to the brain, is about 1.5 orders of magnitude. This 45 hand, compensates for colors present in the ambient 

view is presented, for example, by Creutzfeldt et al. in Rumination automatically so that skin tones, for exam- 

the article entitled "Darkness induction, retinex, and P^, appear the same to the human viewer even under 

cooperative mechanisms in vision," Exp. Brain Res., different lighting conditions. Further, the color 

volume 67, p. 270, 1987. constancy corrections made by the human central ner- 

The retina must cope with a dynamic range mismatch 50 vous system do not noticeably create halo distortion, 

similar to the mismatch, discussed below, of conven- The growing use of video imaging processes uv 

tional artificial sensors and display media, and so must creases the need for real time or near real time unple- 

provide compensating mechanisms. Photographers and mentations of brightness and color constancy correc- 

videographers who are not aware of this fact often think tion techniques and, in particular, implementations 

that there is something wrong with the camera when, in 55 which do not noticeably distort the images, 

viewing the captured image, there is a strong coloration Conventional systems have difficulty in properly 

or deep shadows due to Ulumination not observed when reproducing images ffluminated with widely varying 

the scene was being captured. levels of illuminations. Image sensing processes, or de- 

In fact, however, the brain corrected for the illumina- lectors, commonly have a greater range of sensitivity 
tion at the time of image capture using range compres- 60 than processes that are used to present images. Paper 
sion techniques, but the camera recorded the true scene based processes such as photographic, ink, and electro- 
and the display media could not faithfully reproduce it. static printing and electronic processes that rely on 
Thus it has been desirable to interpose a range compres- cathode ray tubes— or CRTs, liquid crystal dis- 
sion process between image capture and presentation in. plays— or LCDs, electroluminescent displays— or 
order to display recorded images faithfully to the 65 ELDs, plasma displays, and etc., typically cannot pro- 
human viewer, that is, to display recorded images so duce the dynamic range in presentation that is captured 
that they appear in the same way they appear to the by photographic systems or electronic imaging devices 
viewer while being captured. This process may be such as charge-coupled-devices — or CCDs, infra-red or 
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IR devices, charge injection devices—or CIDs, and the may therefore not be simultaneously displayed on a 

like. monitor having a display range of only about 1.5 orders 

In such conventional systems, the minimum level that of magnitude between the brightest and darkest por- 

can be sensed is usually set by the noise level inherent in tions of the displayed image. 

the device. The maximum level that can be sensed in a 5 A well known photographic darkroom technique 

CCD camera, for example, is typically more than 3 called dodging has long been used to cope with the 

orders of magnitude greater than the minimum or noise dynamic range mismatch between a film negative and 

level of the camera. That is, there is about a factor of the photographic print paper. If a photographic nega- 

1000 range of image intensities that can be sensed in a tive of an image including a full-range of intensities is 

CCD camera. However, conventional video monitors 10 printed without special processing, some image portions 

used to view images detected by CCD cameras can may be printed as too bright or too dark, depending on 

often only display intensities covering about 1.5 orders the exposure time used in making the print. If the print 

of magnitude, that is, there is only about a factor of 30 is too dark in places and the exposure is shortened, other 

range of image intensities that can be displayed on such regions will be too light. Similarly, if the exposure time 

monitors. 15 is lengthened to darken areas that are too light, the final 

There are two particularly troublesome problems printed image will include areas that are too dark, 
that occur with dynamic range mismatch between the An experienced photographer, however, may corn- 
image sensor and the display device, that as, when dis- pensate by dodging- In this process, the print is made so 
playing an image detected with a wide-range sensor on that bright areas are printed properly. In areas which 
a limited range monitor or other medium. First, since 20 would otherwise be printed too dark, the exposure is 
the sensor signal must be scaled down for display, the limited by moving an opaque object such as a properly 
incremental contrast of the final result is often poor. shaped card or other obstruction between the print 
Detail in the original scene is displayed at low contrast. paper and the exposing light. The movement of the 
Second, important detail may be displayed near the top opaque card blocking the exposure light reduces the 
or bottom of the monitor range, lost in brightness or 25 total light, and therefore the exposure, in the darker 
darkness. This problem is especially obvious in scenes in areas of the image. Dodging may be thought of as an 
which one area is much brighter than the rest of the equivalent of subtracting a low-pass filtered version of 
scene. the image from the dark regions. Detail is preserved and 

It is well known that the reflectances of natural ob- the intensity of the image area is kept within the range 
jects tends to vary over approximately a factor of 30 of the print paper. If the photographer is not careful and 

twenty. That is, the ratio of intensities across an image inadvertently interferes with exposure light applied to a 

due to the reflectances of the objects in the image varies light region around the dark region, a halo may be 

from the lowest intensity to the highest intensity so that created around the dark region in the final print, 

the highest intensity object is about twenty times more In a related photographic development technique 

luminous than the lowest intensity object in the image. 35 known as masking, a positive transparency is made of a 
This physical phenomena is called herein the Rule of slightly blurred version of the image and inserted in the 

Twenty. path of the exposure beam during development The 

Conventional display media, such as video monitors, blurred positive forms a mask to diminish exposure of 

are often capable of displaying image intensity ranges the print paper in dark regions. Subtraction of this mask, 

greater than required by the Rule of Twenty, typically 40 which is a scaled and low-pass filtered version of the 

such monitors can display output ranges that span about image, reduces the range of the image to match that of 

a factor of thirty. the print paper. Masking also increases the incremental 

However, lighting differences across a scene often contrast in the final print because it is a form of high- 
introduce a greater range of image intensities across the pass filtering. 

scene than caused by the reflectance variation. That is, 45 Unlike the dodging process, the mask which is sub- 
image intensity is equal to the product of illumination tracted does not have sharply defined borders between 
intensity and reflectance. The range of common illumi- regions. Halos are therefore created by masking that are 
nation intensities across a scene are not limited by the often objectionable near large changes in image bright- 
Rule of Twenty. ness. 

In an outdoor scene, for example, the amount of light 50 In the electronic image processing analog to masking, 

reflected from an object that is partly lit by direct sun- a blurred, scaled version of the image is subtracted from 

light and partly in shadow may vary by a factor of 100 the original to scale the range of the final result to fit 

or more even though the portion of this range resulting that of the display medium. The blurred, scaled version 
from reflectance changes does not exceed the Rule of of the image may be obtained optically or electroni- 

Twenty. Conventional image sensors can capture this 55 cally. Near sharp borders or sharp changes of intensity, 

range, but most conventional display media cannot this method produces halos that can be objectionable 

faithfully reproduce it, and may exceed the dynamic range of the presentation 

Flames, for example, will usually overload a monitor medium. To reduce the halo effect, the amount of the 

range unless the input is scaled by, for example, scaling low-pass version that is subtracted varies adoptively 

a sensor input signal electronically, reducing the iris 60 with the magnitude of the spatial derivative of the im- 

aperture or shortening the pixel exposure time. With the age. That is, the degree of masking is adjusted in accor- 

reduced sensitivity of the sensor input, though, it is hard dance with the magnitude of the difference between the 

to see more than a silhouette of what is burning. Impor- sharp and blurred versions of the image. This technique 

tant details such as numbers on a burning vehicle, for of varying the degree of masking is called adaptive 

example, may be lost in darkness. 65 filtering herein. 

In this example, the flames and vehicle numbers are In general, dodging has the advantage over adaptive 
separated in intensity by substantially more than one or filtering for image processing for presentation, because 
1 .5 orders of magnitude. The flames and vehicle number dodging compresses the range of the image to fit that of 
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the presentation medium and preserves the original weighted mask, selecting a predetermined level of in- 

detail contrast uniformly within regions. Adaptive fil- tensity difference related to the intensities above which 

tering does not preserve the original detail as well as the spatial weighting is non-linear and below which the 

dodging. In regions of high contrast texture, for exam- spatial weighting is linear and subtracting the mask 

pie, adaptive filtering changes the image very little. 5 from the image to form a composite image whereby 

Also, the techniques used for overshoot correction in adverse effects of variations in the intensities in the 

adaptive filtering do not enhance the detail near high image are reduced in the composite image, 

contrast edges. In a further aspect the present invention provides a 

Techniques for adaptive filtering are well known, and method of applying range compression to an image by 
are, described for example by William F. Schreiber in 10 applying an image to an analog resistive grid to gener- 
the article entitled "Image Processing for Quality Im- ate a spatially weighted image mask therefrom, adjust- 
provement", Proceedings of the IEEE, Vol. 66, pp. mg the resistive grid to cause image segmentation in 
1640-1651, 1974. In such techniques, low-pass filtered response to differences in image intensities, such differ- 
image gradients serve as an adaptive modifier of the being in a range to be compressed, and subtract- 
low-pass filtered images before subtraction from the 15 mg ^ f rom ^ i magc t0 pro duce a range 
original image, in order to diminish the haloing that compressed composite image, 
would otherwise result from the high-pass filtering step. ^ a further ^ prcscnt invention provides an 

Conventional image processing techniques for color ^ proccssing systcm including an analog smoothing 

and brightness constancy, that is color and Rumination ^ means for ,^ mts of a spcctral component 

mtensity corrections, such as dodging and adaptive 20 of m ^ tQ ^ rclated mts m ^ ^ 

filtering may not be suitable for real-time video image smoothin ^ d t0 form a spcctrftJ component mask, 

processing. Dodging is ^performed ^manually for photo- means for g predetermined , eV el of image 

graphic processing. AoapUve and high pass filtering mtensi difference below which the spatially related 

^ b ^ I ? B W d ff^ P T?° 18 „ points are linearly weighted and above which the spa- 
but have been limited in size of the image area that can 25 ^ k * £ non-linearly weighted in Uie 
be processed with convenient amounts of computing \ , ^ 4 , r L j «T 
power and memory storage spectral component mask, means for forming a drfferen- 

A substantial improvement in image processing, be- ^ m P° n <f mask corresponding to 

yond that available with conventional digital tech- • d ^™ nt ****** of ™9* "f mcans 

niques, which is suitable for real time video image pro- 30 f or ^tracting the spectral component masks from the 

cessing is shown in U.S. Pat. 4,786,818, Mead et. al. nna « e t0 .\ orm » e™Ponte 

which discloses an integrated image processor, known . ]» a stdl further "P 6 *- ^ P resenl f vention pro- 
as a spatially-weighted analog or resistive grid, for pro- "»p Processing system an analog smoothing 
cessing black and white electronic images. The Mead means fo 5 ^pplymg image intensities to the analog 
resistive grid is a low-power analog chip that includes 35 smoothing grid to form a spatiaUy weighted mask, 
image sensors and processors on the same chip and does means for controlling a saturation level related to the 
not require substantial storage memory. The grid per- ^tensities between points in the analog smoothing grid 
forms high-pass filtering of the image and so suffers above which s P atial weighting is non-linear and 
from haloing effects. below which the spatial weighting is linear, and means 

What is needed therefore is a brightness and color 40 for subtracting the mask from the image to form a corn- 
constancy correction technique which permits images P° SIte una g e m which adverse effects of variations in 
to be recorded and/or displayed in a manner similar to fte intensities in the image have been reduced, 
that as perceived by the human viewer at the time of In a sti11 further aspect, the present invention pro- 
image capture which can conveniently be accomplished apparatus for applying range compression to an 
in real or near real time without haloing or other sub- 45 ™ a S e including means for applying an image to an 
stantial distracting distortions. analog grid to generate a spatially weighted image mask 

therefrom, means for adjusting the grid to cause image 

SUMMARY OF THE INVENTION segmentation in response to differences in image intensi- 

The preceding and other shortcomings of the prior ties, the differences being in a range to be compressed, 

art are addressed and overcome by the present inven- 50 and means for subtracting the image mask from the 

tion that provides a method of image processing by image to produce a range compressed composite image, 

applying points of a spectral component of an image to These and other features and advantages of this in- 

spatially related points in an analog smoothing grid, vention will become further apparent from the detailed 

spatially weighting the spatially related points to form a description that follows which is accompanied by one 

corresponding spectral component mask, the spatial 55 or more drawing figures. In the figures and description, 

weighting being linear for differences in image intensity numerals indicate the various features of the invention, 

between neighboring points in the spectral component like numerals referring to like features throughout both 

below a predetermined level related to the image inten- the drawing figures and the description, 

sities in the component and non-linear for differences in _ p .„ nPQri} TT>TTnNI nv to P np A vr Q 

image intensity between neighboring points in the spec- 60 BRIEF DESCRIPTION OF THE DRAWINGS 

tral components above the predetermined level, form- FIG. 1 is a schematic illustration of a segment of * 

ing a second, differentially scaled, spectral component resistive grid 10 according to the present invention, 

mask and subtracting the differentially scaled spectral FIG. 2 is an enlarged schematic illustration of grid 

component masks from the image to form a composite node I/O circuitry 34, depicted in FIG. 1. 

image. 65 FIG. 3 is an illustration of the segment of resistive 

In another aspect the present invention provides a grid 10 depicted in FIG. 1 in which the amplitudes of 

method of image processing by applying image intensi- signals applied to grid nodes are displayed along the 

ties to an analog smoothing grid to form a spatially Vertical or "Z" axis. 
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FIG. 4 is a schematic illustration of color constant particular image. The segment of resistive grid 10 

video system 50 utilizing grids 52, 54 and 56, in accor- shown in FIG. 1 would then represent a portion of the 

dance with the present invention. lower left hand corner of that image, including pixels in 

FIG. 5 is a schematic illustration of a halo corrected columns 0 through 2 of rows 509 through 511. Each 
color constant video system 74 embodiment of color 5 pixel or grid node 12 could then be identified or ad- 
constant video system 50 shown in FIG. 2 above using dressed by its row and column number, such as grid 
a pair of resistive grids in each color channel. node 510,1 shown in FIG. 1 

FIG. 6 is a schematic illustration of derivative node Each grid node 12 is interconnected to its neighbor- 
input channel 102 for one grid node of color transition tog grid nodes by grid element 14 which may be a resis- 
detection grid 78 used in halo corrected color constant 10 tor, capacitor, inductor or other serial element In a 
video system 74 as shown in FIG. 5. practical embodiment of a resistive grid, the physical 

FIG. 7 is an illustration of derivative node input chan- elements interconnecting each node may conveniently 

nel 104, an alternate embodiment of derivative node be controllable elements, such as MOSFET transistors, 

input channel 102, which can be used in color constant controlled to provide the appropriate resistance value, 

video system 50, shown in FIG. 2, to provide halo cor- 15 By way of illustration, grid node 510,1 is connected 

rected color constancy using only one modified resis- to grid node 510,2 via grid element 14, such as resistor 

tive grid per channel. 16. For the purposes of this description, all grid nodes 

FIG. 8 is a schematic illustration of integrating or 12 are assumed to represent points in the color image to 

accumulating array 106 which can be used in lieu of the be corrected and all grid elements 14 are assumed to be 

more complex resistive grid 10 shown in the previous 20 resistors 16 of equal value. 

figures. Each grid node 12 is connected to an input informa- 

FIG. 9 is an illustration of a single grid node element tion channel by grid node I/O circuitry 34, a represen- 

within three line sample and hold 154 which may be tative example of which is shown connected to grid 

added to integrating array 106 for use as a substitute for node 510,1 and is described in more detail with refer- 

color transition detection grids 78, 80 and/or 82 as 25 ence to FIG. 2 below. 

shown in FIG, 5. FIG. 2 is an enlarged schematic illustration of repre- 

F1G. 10 is an illustration of three line sample and hold sentative grid node I/O circuitry 34, depicted in FIG. 1. 

155, a modified version of three line sample and hold Grid node I/O circuitry 34 includes node input channel 

shown in FIG. 9, which may be added to integrating 36 through which a signal representing a color value of 

array 106 for use as a substitute for grids 52, 54 and/or 30 a pixel in an image to be corrected is applied to grid 

56 as shown in FIG. 4 when modified in accordance node 12. Grid node I/O circuitry 34 also includes node 

with derivative node input channel 104 as shown in output channel 38 through which a signal representing 

FIG. 7. a color value of a pixel in a smoothed image is applied 

FIG. 11 is a schematic illustration of a single node of from grid node 12 to other circuitry as described below, 

saturable resistive grid 200 illustrating lateral element 35 for example, with regard to FIG. 4. 

FET pair 205, grid element 242 and bias circuit 210. Grid element 42 may be the same as grid element 14, 

FIG. 12 is a schematic illustration of bias circuit 210 such as resistor 16, and have the same value as all other 

and grid element 242, together with the FETs surround- grid elements or a different value depending upon the 

ing the node, as shown in FIG. 11. application. 

FIG. 13 is a graph illustrating Id as a function of Vds 40 Node output channel 38 provides a high impedance 

for various levels of RBIAS. output connection for detenxiining the corrected color 

FIG. 14 is a block diagram illustration of constancy value on grid node 12 without changing its value such 

correction system 250. as amplifier 44. 

„ 0 ™ TTVT . T ~ XT " -™ The operation of resistive grid 10 will now be de- 
DETAILED DESCRIPTION Or THE 4J ^ with rcgard t0 ^ following example in which 
PREFERRED EMBODIMENTS thc ovcrabundancc of rcd light, if any, is to be deter- 
In the following detailed description of the preferred mined using a common color constancy algorithm. This 
embodiments of the present invention, the use of modi- algorithm is similar to the process proposed by Dr. 
tied resistive grids for color constancy corrections will Edwin H. Land and John J. McCann, described in the 
be described with regard primarily to FIGS. 1 through 50 article entitled "Lightness and Rctinex Theory", Jour- 
10, followed by the description with regard primarily to nal of the Optical Society of America, Vol. 61, January 
FIGS. 11 through 14 of the use of a modified resistive 1971. 

grid including saturable elements for brightness and In accordance with the invention, a composite of 

color constancy correction. spatially extended estimates of the average value of the 

FIG. 1 is a schematic illustration of a segment of 55 spectral content of each point in the image is formed in 

resistive grid 10 constructed in accordance with the accordance with a weighted spatial average. That is, in 

present invention. The use and operation of resistive this example, the intensity of red at each grid node 12 

grid for color constancy correction will be described will be estimated from the average of the intensity of 

below, after a detailed description of its configuration. red at that grid node plus a spatially weighted average 

As shown in FIG. 1, each grid node 12 represents a 60 of the intensity of red at all other grid nodes 12. The 

point in one color channel of a color image to be cor- weighting used is a function of distance, that is, the 

rected for color constancy. In a video imaging system, further from the node of interest, the lower Weight 

each grid node 12 might represent a particular pixel in given to the red intensity of another node, 

the image, each pixel being shown in its proper relation- In particular, the intensity of red light of each pixel in 

ship to surrounding pixels. 65 the image to be corrected is applied to node input chan* 

In a typical video image, for example, a 512 row by nel 36 of grid node I/O circuitry 34 associated with the 

512 column matrix array of pixels might be used to grid node 12 which represents that pixel. A signal repre- 

reprcsent an image frame in each color channel of a senting the intensity of red light for each pixel is there- 
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fore held in the appropriate sample and hold circuitry Color constant video system 50 includes a conven- 

40 associated with the corresponding grid node 12. tional video image capture system such as videocam 58 

FIG. 3 is an illustration of the segment of resistive and a conventional video image display system such as 

grid 10 depicted in FIG. 1 in which the amplitudes of color monitor 60. Color constant video system 50 uses 

signals applied to grid nodes along row 510 and along 5 three color channels, red, green and blue, to capture and 

column 1 as a result of the color value signal applied to reproduce a color image. Other color schemes could be 

grid node I/O circuitry 34 associated with grid node used, with three or more channels, as long as the colors 

510,1 are displayed along the vertical or "Z" axis. were selected so that a full color spectrum could be 

The amplitude of the signal at grid node 510,1 result- reproduced by combining the color channels, 

ing from the color value signal applied to grid node I/O 10 In addition to conventional videocam 58 and color 

circuitry 34 is a function of the relationship between monitor 60, color constant video system 50 includes 

grid element 42, shown in FIG. 2 and all other grid color correction subsystem 62. Videocam 58 is shown 

elements 14 along row 510 and column 1. The ampli- to include nonlinear scaling system 59 the operation of 

rude at grid node 510,1 is a maximum, while the ampli- which will be described below in greater detail with 

tudes at other grid nodes decrease monotonically with 15 respect to FIG. 12. The following description of the 

distance from the maximum. The shape of the decreas- operation of color correction subsystem 62 is applicable 

ing amplitude pattern or profile or transfer function to videocam 58 whether or not non-linear scaling sys- 

may be changed by altering grid node interconnections. tern 59 is included and operable therein. 

In the example shown in FIG. 3, the amplitudes of the Color correction subsystem 62 operates in real time 

voltages at the grid nodes along row 510 are depicted 20 or near real time to improve color constancy by operat- 

by dotted curve 46 while the amplitudes of the voltages ing on the three color channels to reduce the effects of 

at the grid nodes along column 1 are depicted by curve the color of the ambient lighting. In general, color con- 

48. Curves 46 and 48 therefore represent spatially ex- stancy is achieved by determining a spatially extended 

tended estimates of spectral content for the image point and weighted averaged estimate of spectral content for 

represented by grid node 510,1. 25 each pixel in each of the three color channels and re- 

The discussion above was related to the amplitudes of moving the resultant composite. In other words, differ- 

signals along the row arid column associated with a entially scaled spectral component masks are formed by 

single grid node to which a color value signal was ap- application spectral component images to analog 

plied. In the actual operation of resistive grid 10, color smoothing grids and combined with the image, by for 

value signals would be applied to all grid nodes substan- 30 example subtraction, to form a composite image, 

tially simultaneously, in real or near real time. That is, in Video switching network 64 receives the red, green 

video imaging as an example, in the course of a raster and blue color channels from videocam 58 and applies 

scan of one frame, color value signals would be applied them to red, green and blue grids 52, 54 and 56 respec- 

in turn to all sample and hold circuits 40, but each grid tively. Each pixel in each color channel of the color 

element 42 would apply the color value signal held in 35 image from videocam 58 is applied to an appropriate 

the corresponding sample and hold circuitry 40 node input channel 36 corresponding to the assigned 

throughout the raster scan. In other color imaging sys- grid node in the appropriate color correction grid. Each 

terns, the color value signals may be applied simulta- resistive grid forms an image of the spatially weighted 

neously to all grid nodes. The signal amplitudes de- average of the corresponding color signal which is 

tected by node output channel 38, shown in FIG. 2, for 40 provided by each node output channel 38 back to video 

each grid node is a composite of the spatially extended switching network 64. 

estimates of the spectral content at all grid nodes In each channel, the original image from videocam 58 
formed by summing of the signals at each grid node is then modified in accordance with the composite of 
resulting from the signals applied to all grid nodes. In spatially extended estimates of spectral content, or spec- 
this manner, resistive grid 10 may be said to perform a 45 tral mask, by subtraction of the spatially weighted aver- 
low-pass filtering operation on the image. age image from video switching network 64 in red, 
The color value signal input into each node output green and blue image subtracters 66, 68 and 70. The 
channel 38 is a spatially weighted average of the color resultant color constancy improved video image may 
value signals applied to all nodes in resistive grid 10. It then be displayed on monitor 60 as image 72. 
may be helpful to visualize the spreading effect of a SO Referring now again in detail to FIG. 4, the image is 
color value signal applied at each grid node as a tent like sent to grids 52, 54 and 56 for low-pass filtering so that 
outline, with the tent pole for that color value signal at upon subtraction of the grid outputs by way of subtract- 
that grid node. As noted above, the shape of the ampli- ors 66, 68 and 70, the resultant high-pass filtered image 
rude profile or transfer function, or spatially extended has the proper balance of upward-going and down- 
estimate of spectral content, may be different for differ- 55 ward-going overshoots. Conventional video sensors, 
ent applications. A dome shaped or other monotoni- such as videocam 58, often include non-linear scaling 
cally decreasing amplitude profile, with a maximum at system 59 as shown in FIG. 4, so that an appropriate 
the grid node related to the spectral content or color nonlinear scaling correction is applied before low pass 
value signal being smoothed, may be used. In any event, filtering when color correction subsystem 62 is con- 
the compositing of the estimates occurs substantially 60 nected thereto. 

simultaneously or in real time to provide a grid of spa- Image 72 is color constancy corrected in that within 

tially smoothed or averaged values of estimates of spec- each color area or segment, color from the ambient 

tral content formed from the applied color value sig- illumination on average has been removed. However, 

nals. image 72 includes a substantial image degrading error, 
FIG. 4 is a schematic illustration of a color constancy 65 called haloing. In particular, at each major color change 

video system 50 utilizing grids 52, 54 and 56, each of boundary, the color constancy correction is inaccurate, 

which may be constructed in accordance with the de- adding a halo of an inappropriate color on both sides of 

scription of resistive grid 10 shown in FIG. 1 above. a sharp color transition. 
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FIG. 5 is a schematic illustration of a halo corrected 
color constant video system 74 utilizing grids 52, 54 and 
56 as well as videocam 58 and color monitor 60 de- 
scribed above with respect to color constant video sys- 
tem 50 shown in FIG. 2. In place of color correction 
subsystem 62, however, halo corrected color constant 
video system 74 includes color correction subsystem 76. 
Color correction subsystem 76 processes red, green and 
blue color channels from videocam 58 through video 
switching network 64 to develop spatially weighted 
color surround images in grids 52, 54 and 56. The sig- 
nals subtracted from the original three channels of color 
image by red, green and blue image subtracters 66, 68 
and 70 are modified versions of the spatially weighted 
outputs from grids 52, 54 and 56. They are modified in 
accordance with image gradients. In particular, they are 
modified in accordance with the rate of change of illum- 
ination in the vicinity of each pixel. 

The subtracted signals are the product of the outputs 
of grids 52, 54 and 56, and the outputs of color transition 
detection grids 78, 80 and 82, formed in multipliers 84, 
86 and 88. Each color transition detection grid 78, 80 
and 82 receives the same input from video switching 
network 64 applied to the corresponding grids 52, 54 2J 
and 56. The outputs of color transition detection grids 
78, 80 and 82 however correspond to estimates of local 
image gradients, in particular, the degree of "edginess" 
or rate of change of color within each channel at each 
grid node. 

Each pixel in each of these outputs represents the 
degree of color change within each channel in the im- 
mediate vicinity of the grid node related to that pixel. 
Color transition detection grids 78, 80 and 82 may there- 
fore be considered to be low-pass filtering, the deriva- 35 
tives of the images that are low-pass filtered by grids 52, 
54 and 56. 

The configuration of each color transition detection 
grid 78, 80 and 82 are substantially identical to each 



The outputs from differential magnitude detectors 90, 
92, 94 and 96 are combined in average or maximum 
detector 98 to determine the gradient value to be ap- 
plied to grid element 42 connected to grid node 510,1. 
Average or maximum detector 98 may be configured to 
average the input signals, select the largest value or 
perform any other convenient operation for determin- 
ing a value related to the degree of color transition 
between the grid node of interest and its neighbors, that 
is, a local image gradient 

The resultant differential signals are smoothed and 
spatially averaged by the operation of the intercon- 
nected grid nodes in the same manner as described 
above with reference to resistive grid 10 shown in 
FIGS. 1 and 2 above. As discussed above with regard to 
the operation of halo corrected color constant video 
system 74 shown in FIG. 5, the smoothed, spatially 
averaged differential color value signals from each pixel 
in each channel are used to multiply the outputs of grids 
20 52, 54 and 56 before those spectral mask signals are 
subtracted from the original color image in each color 
channel. 

The result is that in halo corrected color constant 
video system 74, the color constancy corrections are 
applied in each channel in a manner reflecting the image 
gradient, the degree of local edginess or color transi- 
tion, as well as the spatially weighted average illumina- 
tion. In the particular example described in detail with 
reference to FIGS. 5 and 6, the color constancy correc- 
tions are weighted in favor of corrections determined 
near the color transition areas, but as a result of smooth- 
ing, the entire image in each channel is color constancy 
corrected for illumination color. 

FIG. 7 shows derivative node input channel 104, an 
alternate embodiment of derivative node input channel 
102 which can be used in place of node input channel 36 
to modify grids 52, 54 and 56 in color constant video 
system 50, shown in FIG. 2, to provide halo corrected 
color constancy using only one such modified resistive 



30 



other and differ from grids 52, 54 and 56 only with 40 grid* per channel. In color constant video system 50 



regard to node input channel 36. FIG. 6 is a schematic 
illustration of derivative node input channel 102 for grid 
node 510,1 of color transition detection grid 78 used in 
halo corrected color constant video system 74 shown in 
FIG. 5. 

The basic configuration of color transition detection 
grid 78 is the same as the configuration for resistive grid 
10 shown in FIG. 1 except that the serial connection 
between sample and hold circuitry 40 and grid element 
42 in each code input channel 36, as shown in FIG. 2, 
has been broken. The value held in sample and hold 
circuitry 40 associated with grid node 510,1 is com- 
pared with the color value signals held in the sample 
and hold circuitry 40 associated with each neighboring 
grid node. 

In particular, as shown in FIG. 6, the color value in 
sample and hold circuitry 40 in image gradient or deriv- 
ative node input channel 102 associated with grid node 
510,1 is compared to the equivalent value in sample and 



45 



50 



55 



using such modified resistive grids, the color constancy 
corrections are determined primarily at color transition 
zones and then smoothed across the image in each color 
channel and applied to the entire image. 

FIG. 7 is a schematic illustration of derivative node 
input channel 104 for grid node 510,1 of a modified 
color transition detection grid 52 used in color correc- 
tion subsystem 62 to provide a halo corrected version of 
color constant video system 50 shown in FIG. 4. 

The basic configuration of modified grid 52 is the 
same as the configuration for color transition detection 
grid 78 shown in FIGS. 5 and 6 except that the value 
held in sample and hold circuitry 40 associated with 
grid node 510,1 is applied to adjustable grid element 100 
instead of grid element 42. The admittance of adjustable 
grid element 100 is controlled in accordance with a 
comparison of the color value signal applied to sample 
and hold circuitry 40 of grid node 510,1 with the color 
value signals held in the sample and hold circuitry 40 



hold circuitry 40 associated with grid nodes 510,0, 60 associated with each neighboring grid node. 



510,2, 509,1 and 511,1 in differential magnitude detec- 
tors 90, 92, 94 and 96, respectively. Each such differen- 
tial magnitude detector operates to determine an image 
gradient at grid node 510,1, the relationship between 
color values at grid node 510,1 and its neighboring grid 65 
nodes by, for example, determining the absolute value 
of the average or maximum value of the difference 
between such values. 



In particular, as shown in FIG. 7, the color value in 
sample and hold circuitry 40 in derivative node input 
channel 104 associated with grid node 510,1 is applied 
to adjustable grid element 100. In addition, the same 
color signal is compared to the equivalent value in sam- 
ple and hold circuitry 40 associated with grid nodes 
510,0, 510,2, 509,1 and 511,1 in differentia! magnitude 
detectors 90, 92, 94 and 96, respectively. Each such 
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differential magnitude detector operates to determine entire image has been integrated with simultaneous 
the relationship between color values at grid node 510,1 weighting in this manner, the image color may be nor- 
and its neighboring grid node by, for example, deter- malized by subtracting the spatial average from the 
mining the absolute value of the difference between input signal in each channel. That is, after integration, 
such values. 5 integrating amy 106 may be substituted for resistive 

The outputs from differential magnitude detectors 90, grid 10 to form a color constancy system. In particular, 
92, 94 and 96 are combined in average or maii^ m integrating array 106 may be substituted for each of the 
detector 98 to determine the admittance of adjustable grids 52, 54 and 56 as shown in FIG. 4 to form color 
grid element 100 connected to grid node 510,1. Average constant video system M* 

or maximum detector 98 may be configured to average 10 During operation of integrating array 106 as shown in 
the input signals, select the largest value or perform any FIG. 8, the color value signal related to each grid node 
other convenient operation for detenmning a value is applied, one at a time during a point by point scan 
related to the degree of color transition between the such as a raster scan, from videocam 58 to common 
grid node of interest and its neighbors such as a value input bus 114 by video switching network 116. Corn- 
proportional to the degree of color transition between 15 mon input bus 114 is connected to row and column 
the neighboring nodes of the grid node of interest. multipliers associated with each grid node. In particu- 

The resultant differential signals are smoothed and lar, row and column multipliers 118 and 120, respec- 
spatially averaged by the operation of the intercon- tively, arc associated with grid node 509,0 while row 
nected grid nodes in the same manner as described and column multipliers 122 and 124 and row and col* 
above with reference to resistive grid 10 shown in 20 umn multipliers 126 and 128 are associated with grid 
FIGS. 1 and 2 above. nodes 510,1 and 511,2, respectively. 

The result is that in halo corrected version of color The multiplications relating to each grid node may be 
constant video system 50 using derivative node input performed in any order, or simultaneously, but are 
channel 104, the color constancy corrections are ap- shown in the indicated order for convenience of discus- 
plied in each channel in a manner reflecting the degree 25 sion. 

of local "edginess" or color transition, as well as the When, for example, the color value signal related to 
spatially weighted average illumination. In the particu- grid node 510,1 is applied to common input bus 114, 
lar example described in detail with reference to FIGS. column spatial weighting profile or transfer function 
2 and 7, the color constancy corrections are determined 132 stored in row weighting subsystem 108 is indexed so 
primarily at color transition zones and then smoothed 30 that maximum value 130, representing the tent pole in 
across the image in each color channel and applied to the earlier analogy, is applied to row multiplier 122. At 
the entire image. the same time, row spatial weighting profile or transfer 

The color constancy correction systems described in function 134 stored in column weighting subsystem 110 
FIGS. 1 through 7 have utilized One or more resistive is indexed so that maximum value 136 is applied to 
grids 10 or the equivalent per color channel. An array 35 column multiplier 124. Maximum values 130 and 136 
of integrating or accumulating elements can be used in may be any value, typically unity, 
lieu of such grids to provide a more easily configured During the raster scan, at the instant of time related to 
equivalent of these color constancy masks. Such an grid node 510,1 the color value signal on common input 
array is used to first sequentially form spatially extended bus 114 multiplied twice by the maximum profile value, 
estimates of spectral content at points in the image and 40 such as unity, is stored in integrating sample and hold 
then to accumulate the estimates to form the composite, 140. At that same instant of time, the same color value 

FIG. 8 shows a segment of integrating array 106 on common input bus 114 is multiplied by next adjacent 
including grid nodes 509,0, 510,1 and 511,2, video profile weight 142 in column multiplier 120 and then by 
switching network 116 and row and Column weighting next adjacent profile weight 144 in row multiplier 118. 
subsystems 108 and 110 which are used in place of the 45 The result is stored in integrating sample and hold 146 
array of grid elements. Integrating array 106 reduces associated With grid node 509,0. Similarly, at that same 
the complexity involved in color constancy correction instant of time, the color value on common input bus 
grid 10 which requires a grid element 14 between each 114 is also multiplied by next adjacent profile weight 
grid node by exploiting the commonalities of the 148 in column multiplier 128 and then by next adjacent 
weighting of the color value signals applied to each grid 50 profile weight 150 in row multiplier 126. The result is 
node. stored in integrating sample and hold 152 associated 

The tent like profile or transfer function of the spa- with grid node 511,2. 
tially extended and weighted estimate of spectral con- Therefore, at the instant of time during the raster scan 
tent is the same for each grid node in a column or row. associated with each grid node, the color value for that 
In a linear grid representation of the image, the profiles 55 grid node may be added to the accumulators, such as 
in columns and rows would be equal to each other. integrating sample and hold 140 associated with each 
Rather than create these profiles independently for each grid node, properly weighted by the weighting profiles 
node in the grid, they may be stored in two dimensional stored in row and column weighting subsystems 108 
row and column weighting subsystems 108 and 110 and and 110. The accumulation of simultaneously spatially 
applied to the color value signals as needed. 60 weighted color values at each grid node occurs at video 

Each grid node only requires three values at one rates so that the values in the accumulators may be 
time; the color value signal, and the row and column subtracted from the original color value in each color 
spatial weighting values. Row and column weighting channel in real time. In particular, color constant video 
subsystems 108 and 110 store the profile so that each system 50 shown in FIG. 4 may be implemented with 
element of the array weights the input as a monotoni- 65 grids 52, 54 and 56 configured in accordance with inte- 
cally decreasing function of distance between the point grating array 106 rather than resistive grid 10, 
in space represented by the element and the point in In the same general manner, one or more of con- 
space represented by the color value signal. Once the stancy correction grids 52, 54, and 56 and/or color 
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transition detection grids 78, 80, and 82, as shown in 
FIG. 5 and/or as modified in accordance with FIG. 7, 
may be configured from one or more integrating arrays 
106. 

Grids 52, 54 and 56 may each be implemented di- 5 
redly from a properly configured integrating array 106 
as shown in FIG. 8. Substitution of integrating array 
106 for color transition detection grids 78, 80 and/or 82 
requires the addition of three line sample and hold 154, 
as shown in FIG. 9, between common input bus 114 and 10 
the column multipliers associated with column 
weighting subsystem 110. Substitution of integrating 
array 106 for constancy correction grids 52, 54 and/or 
56 as shown in FIG. 4 when modified in accordance 
with derivative node input channel 104 as shown in 15 
FIG. 7 requires the substitution of three line sample and 
hold 155 shown in FIG. 10 for three line sample and 
hold 154. 

Referring to FIG. 9, three line sample and hold 154 is 
configured as a three row segment of derivative node 20 
input channel 102 shown in FIG. 6. Video switching 
network 116 applies color value signals to each grid 
node in three line sample and hold 154. After all grid 
nodes in all three lines have received color value sig- 
nals, the values stored in lines 2 and 3 are shifted up to 25 
lines 1 and 2, respectively, and the next row of color 
value signals are applied to line 3. The remainder of 
integrating array 106 operates one raster scan row be- 
hind the raster scanning of video switching network 16. 
The output of integrating array 106 is, of course, in 30 
synch with the raster scan. 

The portion of three line sample and hold 154 shown 
in FIG. 9 includes grid nodes at the intersections of 
columns 0, 1 and 2 with rows 509, 510 and 511. Color 
value signals from common input bus 114 are applied to 35 
sample and hold circuitry 156 associated with each grid 
node. When a color value signal has been applied to a 
particular grid node on line 3 of three line sample and 
hold 154, the grid node associated with the same row in 
line 2 of three line sample and hold 154 may be pro- 40 
cessed and applied to the corresponding column multi- 
plier. The following example of the operation of three 
line sample and hold 154 is based on processing of grid 
node 510,1 into integrating sample and hold 140, as 
shown in FIG. 8 45 

Referring again to FIG. 9, after the color value for 
grid node 511,1 has been applied from common input 
bus 114 to the appropriate grid node in line 3 of three 
line sample and hold 154, the spatial derivative of the 
color value related to grid node 510,1 is determined. In 50 
particular, the color value signals associated with grid 
node 509,1 in line 1, grid nodes 510,0 and 510,2 in line 2 
and grid node 511,1 in line 3 are compared with the 
color value signal associated with grid node 510,1 in 
line 2 by differential magnitude detectors 158, 160, 162 55 
and 164 and combined in average or maximum detector 
166. 

The output of average or maximum detector 166 is a 
measure of the image gradient, the degree of edginess or 
color transition, at the pixel represented by grid node 60 
510,1. The output of average or maximum detector 166 
is applied to the column multipliers shown in FIG. 8. At 
the same time, column weighting subsystem 10 and row 
weighting subsystem 108 are indexed so that maximum 
value 136 and maximum value 130 are applied to col- 65 
umn multiplier 124 and row multiplier 122, respec- 
tively. The. results of these multiplications are stored 
and accumulated in integrating sample and hold 140. 
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After a complete frame has been processed for stor- 
age in integrating array 106 one row late in accordance 
with the above description of three line sample and hold 

154, integrating array 106 provides the spatially 
weighted color value derivative information necessary 
for replacement of color transition detection grids 78, 
80 and/or 82, shown in FIG. 5. 

FIG. 1 is an illustration of three line cample and hold 

155, a modified version of three line sample and hold 
154 shown in FIG. 9, which may be added to integrat- 
ing array 106 for use as a substitute for grids 52, 54 
and/or 56 as shown in FIG. 4 when modified in accor- 
dance with derivative node input channel 104 as shown 
in FIG. 7. In this manner, a complete color constancy 
correction system including halo correcting derivative 
grids may be configured from a single integrating array 
106 per color channel. 

The portion of three line sample and hold 155 shown 
in FIG. 10 also includes grid nodes at the intersections 
of columns 0, 1 and 2 with rows 509, 510 and 511. Three 
line sample and hold 155 operates in the same manner as 
three line sample and hold 154 except that the output of 
average or maximum detector 166 is not applied di- 
rectly to column multiplier 124 as shown in FIG. 8. 

An integrating array 106 when modified with the 
addition of three line sample and hold 154 determines 
the spatially smoothed derivative of the color value 
signal. As shown in color correction subsystem 76 in 
FIG. 5, this spatially smoothed derivative image, similar 
to the outputs of color transition detection grids 78, 80 
and 82, may be multiplied by the spatially weighted grid 
image and then subtracted from the original color image 
to provide an illumination corrected image. 

The complexity of the system may be further reduced 
by combining the spatially weighted average and deriv- 
atives in the same grid as described with regard to de- 
rivative node input channel 104 in FIG. 7. The follow- 
ing description of the additional circuitry in three line 
sample and hold 155 permits the same reduction in 
complexity. 

The output of detector 166 is a measure of image 
gradient at the pixel represented by grid node 510,1. 
The output of average or maximum detector 166 may be 
combined in grid node multiplier 168 with the original 
color value signal for the grid node being processed. 
The output of grid node multiplier 168 is then applied to 
the column multipliers shown in FIG. 8. The resultant 
output of integrating array 106 has therefore already 
been combined with the color value signal before spatial 
weighting. In this way, only one integrating array 106 is 
required per color channel to provide a spectral mask 
for color constancy correction without haloing. 

The methods described above with regard to FIG. 1 
through 10 for brightness and color constancy correc- 
tion may all be considered to be forms of filtering. The 
resistive grid, and the integrating array, provide a con- 
venient parallel processing, analog network for low- 
pass filtering an image. 

Grids 52, 54, and 56 as shown in FIGS. 4 and 5, may 
be implemented as resistive grids as shown in FIG. 1 or 
as integrating arrays as shown in FIG. 8, to provide 
low-power, compact, video-rate filtering. To adap- 
tively modulate the low-pass filtered image before sub- 
traction, a low-pass filtered version of the image gradi- 
ents may be obtained at video rates with the modified 
resistive grid shown in FIG. 6 or the integrating array 
as shown in FIG. 9. These modulating filters are repre- 
sented as grids 78, 80, and 82 as shown in FIG. 5. 
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Alternatively, the low-pass filtering and gradient with node 510,2 and FET 218 associated with node 
modulation may be implemented in a single, low-power, 509,1, are controlled by bias circuits similar to bias 
compact, device that operates at video rates by modify- circuit 210. 

ing the resistive grid as shown in FIG. 7 or the integrat- The input to all such bias circuits is controlled by the 
ing array as shown in FIG. 10 to form filtering grids 52, 5 input signal RBIAS, which therefore controls the 
54, and 56. source-to-gate voltages of all lateral element FET pairs 

As noted above, sensor signals from sensors such as used to interconnect nodes in saturable resistive grid 
videocam 58 shown in FIG. 4 have greater dynamic 200. A particular implementation of bias circuit 210 is 
range than may be displayed on display media such as shown in the implementation of grid element 242 shown 
color monitor 60, requiring that the image as detected 10 in FIG. 12 and disclosed in greater detail in the Mead et 
often be scaled down for the range of the display media. al. patent referenced above, for example, in FIGS. 4a 
The techniques described above with regard to FIGS. 1 and 4c therein. 

through 10 can be used to ameliorate the image degra- Referring now to FIG. 12, grid element 242 associ- 
dation that results from such scaling. ated with node 510,1 is disclosed, together with FETs 

Such scaling is necessary to insure that the sensor 15 206, 212, 214, and 216, as a preferred embodiment of 
signal does not overload the display medium, but often grid element 42 of FIG. 2. The image intensity signal 
resultants in the undesirable lowering of the incremental handled by sample and hold circuitry 40 for node 510,1 
contrast of the image as displayed when compared to is applied to one input of operational amplifier— or op 
the original image. One technique for further reducing amp — 220, the other input of which is connected to its 
the image degradation resulting from such scaling is to 20 output, FET 222 at the input of bias circuit 210, and grid 
preserve the high frequency information in the image node 12 at the common connection of the source termi- 
by high pass filtering. High pass filtering of the image rials of FETs 206, 212, 214, and 216. The INBIAS bias 
may be achieved by subtracting a low-pass filtered ver- signal is applied to op amp 220 to control the effective 
sion of the image from the image before display. resistance of grid element 242 and therefore the current 

This restores some of the incremental contrast that 25 flowing through each lateral element FET pair 205 to 
results from the sensor signal scaling. Simple high pass adjacent nodes in order to cause smoothing, 
filtering, however, results in haloing at large steps in the In bias circuit 210, input FET 222 is one leg of a 
image. As described below with regard now to FIGS. current divider including FETs 224 and 226 in the other 
11 through 14, high frequency information, which leg. Power is applied to the current divider by FETs 
serves along with low frequency information to define 30 228 and 230 and the current divider legs are connected 
the step, is preserved in the low pass filtered image to be to ground through FET 232 the gate input of which is 
subtracted during the high pass filtering process in controlled by RBIAS. In the linear mode of operation 
order to reduce or eliminate the haloing problem in the of saturable resistive grid 200, which is the same as the 
resultant image as displayed. normal mode of operation of resistive grid 10, INBIAS 

In order to preserve the high frequency image infor- 35 and RBIAS are globally adjusted so that grid element 
xnation in the low pass filtered version of the image, 42 and grid element 14 operate in a linear manner, 
resistive grid 10 as shown FIG. 1 may be operated as Referring now to FIG. 13, the drain current Id for 
shown in FIG. 11 to preserve high frequency infonna- each FET of lateral element FET pair 205 is shown as 
tion only at large steps in the image. This is accom- a function of the drain source voltage Vds there across 
plished by providing grid element 14 with a reduced 40 for FET 26 for three different settings of RBIAS. The 
linear range by controlling the bias thereof to control maximum I/>is controlled by the source-to-gate voltage 
the saturation of grid element 14. The low pass filtered for each such FET and therefore by RBIAS. As RBIAS 
version of the image with high frequency image infor- is increased, the maximum available current through 
mation is then preserved only at large step changes in each such FET is increased. For any particular setting 
the image intensity to provide a mask suitable for sub- 45 of RBIAS, where Vds is below tie saturation vol- 
traction from the original image to reduce the range of tage,— or V M1 — of FET 206, a current is caused to flow 
the sensor signal while maintaining or enhancing incre- through FET 206 that is linear with respect to the volt- 
mental contrast without causing haloing. age applied. In this manner, each such lateral element 

Referring now to FIG. 11, an enlarged portion of FET pair 205 acts as a resistor because Vds is linearly 
saturable resistive grid 200, similar to resistive grid 10 50 related to I/>. 

described above with regard to FIG. 1, is shown sur- However, if Id exceeds V M/ , the FET saturates and 
rounding node 510,1. Resistive grid 10, as well as satura- ID reaches its maximum. Any further slight increase in 
ble resistive grid 200, may be implemented as described current for Vds greater than V«/, due for example to 
by Mead et al. in U.S. Pat. No. 4,786,818. In such an the channel length modulation ofthe FET known as the 
implementation, each grid element 14 is configured 55 Early effect, is relatively minor compared to the incre- 
from a series connection of a pair of field effect transis- mental current increase for Vds below V^,, and may be 
tors — or FETs—, shown in FIG. 11 as lateral element ignored for the purposes of the present description. 
FET pair 205. Lateral element FET pair 205 between Grid element 14 may operate as both a linear element 
nodes 510,1 and 510,2 includes FET 206 associated with like a resistor for Vds below V tt/ and as a non-linear 
node 510,1 and FET 208 associated with node 510,2. 60 element as a current source for Vds greater than V„ r . 
The current flowing in series between nodes 510,1 and Saturable resistive grid 200 therefore operates in differ- 
510,2 flows through each FET of lateral element FET ent modes depending upon the relationship between the 
pair 205. actual Vds and Vsot- 

The source-to-gate voltage of FETs 206, 212, 214, In normal or linear operation, saturable resistive grid 
and 216 surrounding node 510,1 is held constant by bias 65 200 acts in the manner described above with regard to 
circuit 210, shown in greater detail in FIG. 12. In a resistive grid 10 and smooths or low-pass filters an array 
similar manner, the source-to-gate voltage of FETs of inputs applied to each node, such as node 510,1, by 
surrounding other nodes, such as FET 208 associated the spreading action of currents through each grid ele- 
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ment 14 as long as Vjjsfor each FET of lateral clement 
FET pair 205 does not exceed V»,. If, however, Vds 
exceeds V Mr , the operation of saturable resistive grid 
200 becomes non-linear at least between the nodes inter- 
connected by the lateral element FET pair 205 in which 
V/xs has exceeded V^. 

For example, if the difference between signal inputs 
applied to a pair of adjacent nodes, such as node 510,1 
and node 510,2, is large enough so that for FET 206 
exceeds V w/r saturable resistive grid 200 is operated in a 
non-linear manner in the book entitled "Analog VLSI 
and Neural Systems," Addtson-Wesley, New York, 
1989, pp. 122-123. 

The relationship between Vds and V mt can be 



10 



intensity at the edge between the bright window and the 
darkened sill. 

Referring now for an illustrative example to the red 
channel of image processing provided in color con- 
stancy correction grid 52 shown in FIG. 4, the grid 
nodes 12 on opposite sides of the step change edge 
would have substantially different intensities of image 
signals applied thereto by the appropriate node input 
channel 36 as shown in FIGS. 2. If the magnitude of the 
step change exceeds the dynamic range of color moni- 
tor 60 used to display the image, the image may be 
scaled by operating color constancy correction grid 52 
in the manner of saturable resistive grid 200 described 
above with regard to FIG. 11 so that lateral element 



changed for any given value of V/jyby simply adjusting 15 FET pair 205 between the nodes surrounding the image 



Vjb/. As is well known in the art, changing Vcs changes 
Vjof. That is, as Vcs is increased, the maximum current 
increases and therefore V^is increased. In accordance 
with the present invention, the relationship between 
\ds and Vjo, is intentionally adjusted by for example 20 
adjusting RBIAS to change Vcs- Changing \cs 
changes V tf/ and causes saturable resistive grid 20 to go 
into saturation. 
For example, referring now to FIG. 14, a nominal 



intensity step reach saturation. 

As noted above, saturable resistive grid 200 may be 
operated to selectably saturate for edges greater than a 
predetermined magnitude, that is, for Vqs greater than 
a predetermined magnitude, by selectively adjusting 
RBIAS to selectively adjust \W Therefore, when V«, 
for each FET in lateral element FET pair 205 is set 
lower than the \jjs applied across by the step change 
edge of the image, segmentation occurs. By adjusting 



setting of RBIAS results in a value of Vcsthat produces 25 RBIAS to cause segmentation at such edges, color con- 



a saturation voltage of V a th An increase in the value of 
RBIAS directly increases the value of Vcs and results 
in 8 greater saturation voltage Vjatit and a still greater 
value of RBIAS increases Vcs still further and results in 
a still greater saturation voltage, V^. 

Saturable resistive grid 200 then operates in a non-lin- 
ear manner when Vds> half the voltage drop across 
lateral element FET pair 205 between any pair of nodes, 
exceeds a predetermined value. 

That is, when there is a large step in image intensity 35 
between two image regions or segments, saturable resis- 
tive grid 200 is allowed to operate in a linear manner 
within each such image region to low-pass filter the 
image therein removing all high frequency components 



stancy correction grid 52 may be operated in a non-lin- 
ear manner so that low pass filtering or smoothing oc- 
curs in the bright and dark regions separated by the 
window sill while retaining the high frequency or step 
30 change information related to the step change in inten- 
sity. When the resultant smoothed image or spectral 
mask is then subtracted in red image subtractor 66, 
image 72 may be displayed on color monitor 60 without 
haloing at the step change edge. 

Since the reflectances of natural objects are known, 
as noted above, to vary in magnitude by no more than 
about a factor of twenty, steps in image intensity indi- 
cating changes of magnitude of intensity that are 
greater than about a factor of twenty are often due 



of the image in such regions. However, at the large step 40 solely or in part to lighting effects such as shadows. 



change in image intensity at the edges between such 
regions, saturable resistive grid 200 is caused to operate 
in a non-linear manner to preserve both the high and 
low frequency information in the filtered image at the 
edge between image regions. 

When the image filtered in this way is subtracted 
from the original image, the range of the resulting signal 
is reduced as compared to the original signal without 
reducing incremental contrast, and therefore preserving 
detail, without introducing halo distortions at such re- 
gion edges in which the large step change between 
regions causes non-linear operation of saturable resis- 
tive grid 200. 

By controllably driving saturable resistive grid 200 



The Rule of Twenty is useful in determining how to 
adjust resistive grid 10 to operate in the manner of satu- 
rable resistive grid 200 to remove lighting effects when 
such a grid is used as correction grid 52 of color con- 
45 stant video system 5 as shown in FIG. 4. Grid element 
42, such as the implementation shown as grid element 
242 in FIG. 12, should be operated in a linear mode. 
This accomplished by choosing the setting for INBIAS 
so that op amp 220 operates in a linear region over the 
50 entire input signal range. 

The sensor noise level, that is, the noise level of 
videocam 58, should then be determined. A noise level 
of about 20 millivolts is typical. Applying a factor of 



twenty to this noise level, in accordance with the Rule 
into non-linear operation for step changes in intensity 55 of Twenty referenced above, indicates that the maxi- 
greater than a predetermined level between regions, mum signal above noise that would be expected to re- 
sensor scaling may be accomplished without excessive suit from changes in image intensity, would produce a 
loss of contrast and detail in individual regions and maximum signal on the order of about 400 millivolts, 
without causing haloing at the edges between regions. Therefore, in accordance with this invention, for a sen- 
In this manner, a video image with both bright and dark 60 sor having an input noise level of about 20 millivolts, 



regions or segments, such as might occur in a video 
image captured in a slightly darkened room including a 
bright sunlit image through a window, may be pro- 
cessed for improved display on a monitor having a 
dynamic range smaller than the range required to dis- 
play both the darkened room and the bright window 
image. In particular, at the window sill in the image for 
example, there will be a large step change in image 



65 



the magnitude or setting of RBIAS should be selected 
so that lateral element FET pair 205 saturates or oper- 
ates in a non-linear region for image intensity step 
changes greater than about 400 millivolts. 

It may be desirable to retain some information about 
lighting, for cosmetic or computational reasons, and so 
lateral element FET pair 205 may be biased for satura- 
tion at steps that are slightly greater than a factor of 
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twenty. However, the point of saturation should not be AGC— technique to assure that the display range of 
set so high that the range of the output device, such as monitor 60 is not exceeded. A typical display monitor 
color monitor 60, is exceeded for a test image contain- such as a CRT has a maximum allowable signal value of 
ing a large step change in image intensity. about 0.9 Volts, after any non-linear scaling correction 

In summary, saturable resistive grid 200 as shown in 5 has been applied by, for example, non-linear scaling 
FIG. 11 smooths through small steps in image intensity system 61. Although non-linear scaling system 59 is 
but not through large steps in intensity, and the degree shown in FIG. 4 as included within videocam 58 to 
of smoothing or the transfer function is variable, being reflect the current practice in the design and manufac- 
set by the ratio of the effective resistance of grid ele- ture of such cameras, in accordance with the present 
ment 42 to the effective resistance of grid element 14 in 10 invention, it is preferable to provide such nonlinear 
its linear range of operation. scaling after the application of non-linear filtering with 

With regard therefore to FIG. 4, for regions or seg- saturable resistive grid 200 as shown in FIG. 14. 
ments of the input image applied by videocam 58 image In addition, it is well known that illumination inter- 
to color constancy correction grid 52 without large step acts multiplicatively with reflectance. That is, in a loga- 
changes in intensity, the grid may be operated in a linear 15 rithm of an image, the illumination and reflectance add. 
region to provide low pass filtering of the image and The illumination may then be subtracted directly, that 
therefore linear smoothing with a transfer function lin- is, a single value for RBIAS may be used to set V Mt and 
ear within the segment. The subtraction of the therefore the intensity level at which segmentation is 
smoothed image in red image subtractor 66 will effect a required to occur. 

high pass filtering of the image, so that incremental 20 Subtraction of a logarithmically scaled, nonlinearly 
contrast is improved. filtered image signal from the logarithmically scaled 

For adjacent regions separated by a step change in original image signal serves to provide an enhanced, 
intensity beyond a predetermined level dictated by the dynamically scaled logarithmic image. The logarithmic 
Rule of Twenty or other level less than the dynamic scaling may then be removed and any other scaling, by 
range of the display device, the grid is operated in a 25 for example non-linear scaling system 61 may then be 
non-linear manner at the edge of such regions. Since the applied. 

range of the input image is reduced by subtraction of the If a logarithmic scaling is not used, or is used after a 
non-linearly filtered image, saturable resistive grid 200 non-linear scaling has been applied, suitable scaling for 
provides a mask which, when subtracted as a spectral V M r must be applied. That is, in a non-logarithmic 
mask for example in red image subtractor 66, achieves 30 processing system, the present invention benefits from 
image contrast enhancement and range reduction simul- adjusting the intensity level at which saturation, and 
taneously. therefore segmentation, occurs as a function of signal 

Therefore, with reference to a particular node of intensity in order to accurately maintain the relative 
saturable resistive grid 200, the transfer function may be contrast levels within each region, 
a linear tent — or dome-shaped profile as described 35 Image enhancement system 250 shown in FIG. 4 is 
above, or the transfer function referenced to that node configured to provide a logarithmically scaled system, 
may be spatially nonlinear, with smooth linear spatial Only one channel of video information will be de- 
weighting segments separated by a discontinuous con- scribed with regard to constancy correction system 250 
tour along which the lateral elements of the grid are for convenience although multiple color channels will 
saturated, due to inputs in the segments on either side of 40 most likely be used to provide color video enhance- 
the contour that result in values across the lateral ele- ment In constancy correction system 250, videocam 58 
ments that exceed the saturation level of the elements. provides the input video signal of the image, without if 

A resistive grid with nonlinear resistors such as satu- possible, any scaling such as by non-linear scaling sys- 
rable resistive grid 200 illustrated in FIG. 11 is a partic- tern 59 shown in FIG. 4. The input video image is first 
ular case of a general class of diffusive substrates capa- 45 applied to logarithmic scaler 254 the output of which is 
ble of providing non-linear filtering. The disclosed de- the logarithm of the input and is applied in parallel to 
vice uses current diffusion for smoothing, with non-lin- saturable resistive grid 200, peak or maximum detector 
ear resistors to restrict flow, but charge flow, heat flow, 256, and subtractor 258. 

and fluid flow are other examples of physical processes Subtractor 258 performs the same function as image 
that could similarly be used to filter an image. Lateral 50 subtracters 66, 68, and 70 perform for the each channel 
elements could be used with such devices to restrict the processed by video switching network 64 in FIG. 2. 
flow of charge, heat, or fluid in order to carry out simi- That is, subtractor 258 is used to subtract the smoothed 
lar, non-linear filtering operations. image mask from the original image so that the resultant 

For example, if charge is stored at each node in an subtracted image may be displayed. Because the image 
array in proportion to the intensity at the corresponding 55 has been scaled by logarithmic scaler 254 for conve- 
point in an image, a copy of the image is stored in the nience, after processing by subtractor 258 the image 
array. If each node is connected to its neighbors by must be returned to linear scaling by antilog scaler 26. 
switches, and the switches are opened briefly, the image Any further non-linear scaling, such as normally pro- 
is smoothed as charge spreads throughout the array. If vided by non-linear scaling system 59 shown in FIG. 4, 
the switches limit the amount of charge that may flow, 60 may now be provided in a subsequent processing step 
then such switches serve the same purpose as the lateral by non-linear scaling system 61. 
element FET pair 205 shown in FIG. 11 and so the As noted above the logarithmically scaled image 
charge read out after such an operation could be used to output from logarithmic scaler 254 is also applied to the 
form a mask for subtraction from the original image in input of saturable resistive grid 200 as described in detail 
accordance with the present invention. 65 with regard to FIG. 11 above. Saturable resistive grid 

Referring now to FIG. 14, constancy correction sys- 200 must be properly adjusted so that segmentation, 
tern 250 provides signal compression of the input image that is, saturation of the lateral element FET pair 205 
from videocam 58 with an automatic gain control — or between nodes on either side of a large step change in 
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image intensity, occurs substantially in accordance with 
the Rule of Twenty. In particular, as noted above, V„, 
for the lateral element FETs is moved by adjustment or 
setting of RBI AS bo that intensities substantially greater 
than about a factor of twenty greater than the effective 
noise level of videocam 56 cause saturation and there- 
fore segmentation to preserve the high frequency com- 
ponents in the smoothed image or mask. 

The output of saturable resistive grid 200 must then 
be linearly scaled for subtraction from the unsmoothed 10 
signal so that the resultant image signal does not exceed 
the maximum allowable signal that can be applied to 
monitor 60. Peak or wftTimnm detector 256 is used to 
compare the rri>T ' lw " Tn signal value of the logarithmi- 
cally scaled image output from logarithmic scaler 254 13 
with a properly scaled maximum allowable signal value, 
VmcM- In the example being described, would be 
the log of 0.9 volts. The output of maximum detector 
256 is compared in error detector 264 with Vmax to 
determine a feedback error signal applied through com- 20 
parator 266 to multiplier 268. 

Multiplier 268 adjusts the output of saturable resistive 
grid 200 in accordance with the factor provided by 
error detector 264 and applies the adjusted output to 
subtractor 258. In addition, the output of multiplier 268 23 
is also applied to maximum detector 270 and minimum 
detector 272 so that the differential value of the 
smoothed image signal intensity may be determined in 
comparator 274 and provided as the other input to com- 
parator 266. In this manner, the output of saturable 30 
resistive grid 200 may be properly scaled so that it is 
exceeded by the unsmoothed image output of logarith- 
mic scaler 254 by Vmax- Therefore, after subtraction in 
subtractor 258, the maximum of the signal intensity of 
the image output applied to antilog scaler 260 is Vmax- 35 

Alternate smoothing and AGC techniques may be 
utilized with the scope of the present invention, but 
scaling the smoothed image mask to be subtracted from 
the unsmoothed image preserves the relative contrasts 
in each region more accurately than simply scaling the 40 
entire image. That is, the entire usable image signal is 
applied to non-linear scaling system 61 and or monitor 
60 at the maximum allowable signal levels by subtract- 
ing a properly scaled mask. The relative contrast levels 
are therefore not distorted within regions but rather 45 
only at the edges of such regions which exceed intensity 
changes on the order predicted by the Rule of Twenty. 

While this invention has been described with refer- 
ence to its presently preferred embodiments), its scope 



subtracting said spectral component masks from said 
image to form a composite image in which the 
maximum image intensity difference is within the 
dynamic range of an image utilization device re- 
sponsive thereto. 

2. The method of claim 1 further comprising the steps 



of: 



adjusting a transfer function of the linear spatial 

weighting; and 
separately adjusting the predetermined level 
3. The method of claim 1 further comprising the step 



of: 



selecting the. predetermined level to reduce the ef- 
fects of illumination of the image in the composite 
image. 

4. The method of claim 1 further comprising the step 



of: 



is not limited thereto. Rather, such scope is only limited 50 with the rule of twenty. 



selecting the predetermined level substantially in 
accordance with the rule of twenty. 

5. The method of claim 1 further comprising the steps 
of: 

applying a logarithmic scale factor to the spectral 
component before said component is applied to the 
analog smoothing grid; 

selecting a fixed value of said difference in image 
intensity between neighboring points as the prede- 
termined level; and 

applying an anti-logarithmic scale factor to the com- 
posite image before further processing or display. 

6. The method of applying range compression to an 
image, comprising the steps of: 

applying an image to an analog resistive grid to gen- 
erate a simultaneously segmented, spatially 
weighted image mask therefrom, the resistive grid 
being adjusted to cause image segmentation in re- 
sponse to differences in image intensities by 
weighting portions of said differences below a pre- 
determined level in a first manner and by weighting 
any portions of said differences above that level in 
a second manner; and 

subtracting the image mask from the image to pro- 
duce a composite image in which the differences 
between the image intensities is range compressed 
to within the dynamic range of an image utilization 
device related thereto. 

7. The method of claim 6 wherein image segmenta- 
tion results from saturation of elements of the grid, 

8. The method of claim 7 wherein the grid is adjusted 
for image segmentation substantially in accordance 



insofar as defined by the following set of claims and all 
equivalents thereof 
What is claimed is: 

1. A method of image processing, comprising the 
steps of: 55 
applying points of a spectral component of an image 
to spatially related points in an analog smoothing 
grid; 

spatially weighting said spatially related points to 
form a corresponding spectral component mask, 60 
said spatial weighting being simultaneously applied 
in a linear manner for portions of such differences 
in image intensity between neighboring points in 
the spectral component below a predetermined 
level and in a non-linear manner for any portions of 65 
such differences above said predetermined level; 

forming a second, differently scaled, spectral compo- 
nent mask; and 



9. The method of claim 6 further comprising the step 



of: 



separately adjusting a transfer function of said spatial 
weighting. 

10. Apparatus for image processing, comprising: 

analog smoothing grid means for linearly spatially 
weighting portions of applied image intensity dif- 
ferences below a predetermined difference level 
and simultaneously non-linearly spatially 
weighting applied image intensities above said pre- 
determined difference level of a spectral compo- 
nent of an image applied to said analog smoothing 
grid to form a spectral component mask; 

means for forming another, differently scaled spectral 
component mask corresponding to a different spec- 
tral component of said image; and 

means for subtracting said spectral component masks 
from said image to form a composite image. 
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11. The invention of claim 10 further comprising: 
means for adjusting a transfer function of the linear 

spatial weighting; and 
means for separately adjusting said predetermined 
level 

12. The invention of claim 10 further comprising: 
means for selecting the predetermined level to reduce 

the effects of illumination of the image in the com- 
posite image. 
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13. The invention of claim 10 wherein said predeter- 
mined level is selected substantially in accordance with 
the rule of twenty. 

14. The invention of claim 10 wherein said means for 
forming a differentially scaled spectral component mask 
further comprises: 

means for applying a logarithmic scale factor to the 
spectral component before said component is ap- 
plied to the analog smoothing grid; and 

means for applying an anti-logarithmic scale factor to 
the composite image before further processing or 
display. 
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